Black-box Model Research Articles

Interpretable Bike-Sharing Activity Prediction with a Temporal Fusion Transformer to Unveil Influential Factors: A Case Study in Hamburg, Germany

Bike-sharing systems (BSS) have emerged as an increasingly important form of transportation in smart cities, playing a pivotal role in the evolving landscape of urban mobility. As cities worldwide strive to promote sustainable and efficient transportation options, BSS offer a flexible, eco-friendly alternative that complements traditional public transport systems. These systems, however, are complex and influenced by a myriad of endogenous and exogenous factors. This complexity poses challenges in predicting BSS activity and optimizing its usage and effectiveness. This study delves into the dynamics of the BSS in Hamburg, Germany, focusing on system stability and activity prediction. We propose an interpretable attention-based Temporal Fusion Transformer (TFT) model and compare its performance with the state-of-the-art Long Short-Term Memory (LSTM) model. The proposed TFT model outperforms the LSTM model with a 36.8% improvement in RMSE and overcomes current black-box models via interpretability. Via detailed analysis, key factors influencing bike-sharing activity, especially in terms of temporal and spatial contexts, are identified, examined, and evaluated. Based on the results, we propose interventions and a deployed TFT model that can improve the effectiveness of BSS. This research contributes to the evolving field of sustainable urban mobility via data analysis for data-informed decision-making.

Frequency domain model-augmented feature space transferable attack

Deep neural networks (DNNs) are susceptible to adversarial attacks, and one important factor is that adversarial samples are transferable, i.e., adversarial samples generated by a particular network may deceive other black-box models. However, existing transferable adversarial attacks tend to modify the input features of images directly without selection to reduce the prediction accuracy in the alternative model, which would enable the adversarial samples to fall into the model’s local optimum. Alternative models differ significantly from the victim model in most cases, and while simultaneously attacking multiple models may improve transferability, gathering numerous different models is more challenging and expensive. We simulate various models using frequency domain transformation to close the gap between the source and victim models and improve transferability. At the same time, we destroy important intermediate layer features that influence the decision of the model in the feature space. Additionally, smoothing loss is introduced to remove high-frequency perturbations. Extensive experiments demonstrate that our FM-FSTA attack generates more well-hidden and transferable adversarial samples, and achieves a high deception rate even when attacking adversarially trained models. Compared to other methods, our FM-FSTA improved attack success rate under different defense mechanisms, which reveals the potential threats of current robust models.

A Meta Algorithm for Interpretable Ensemble Learning: The League of Experts

Background. The importance of explainable artificial intelligence and machine learning (XAI/XML) is increasingly being recognized, aiming to understand how information contributes to decisions, the method’s bias, or sensitivity to data pathologies. Efforts are often directed to post hoc explanations of black box models. These approaches add additional sources for errors without resolving their shortcomings. Less effort is directed into the design of intrinsically interpretable approaches. Methods. We introduce an intrinsically interpretable methodology motivated by ensemble learning: the League of Experts (LoE) model. We establish the theoretical framework first and then deduce a modular meta algorithm. In our description, we focus primarily on classification problems. However, LoE applies equally to regression problems. Specific to classification problems, we employ classical decision trees as classifier ensembles as a particular instance. This choice facilitates the derivation of human-understandable decision rules for the underlying classification problem, which results in a derived rule learning system denoted as RuleLoE. Results. In addition to 12 KEEL classification datasets, we employ two standard datasets from particularly relevant domains—medicine and finance—to illustrate the LoE algorithm. The performance of LoE with respect to its accuracy and rule coverage is comparable to common state-of-the-art classification methods. Moreover, LoE delivers a clearly understandable set of decision rules with adjustable complexity, describing the classification problem. Conclusions. LoE is a reliable method for classification and regression problems with an accuracy that seems to be appropriate for situations in which underlying causalities are in the center of interest rather than just accurate predictions or classifications.

A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges

Graph Neural Networks (GNNs) perform well in community detection and molecule classification. Counterfactual Explanations (CE) provide counter-examples to overcome the transparency limitations of black-box models. Due to the growing attention in graph learning, we focus on the concepts of CE for GNNs. We analysed the SoA to provide a taxonomy, a uniform notation, and the benchmarking datasets and evaluation metrics. We discuss fourteen methods, their evaluation protocols, twenty-two datasets, and nineteen metrics. We integrated the majority of methods into the GRETEL library to conduct an empirical evaluation to understand their strengths and pitfalls. We highlight open challenges and future work.

Active robotic search for victims using ensemble deep learning techniques

In recent years, legged quadruped robots have proved to be a valuable support to humans in dealing with search and rescue operations. These robots can move with great ability in complex terrains, unstructured environments or regions with many obstacles. This work employs the quadruped robot A1 Rescue Tasks UPM Robot (ARTU-R) by Unitree, equipped with an RGB-D camera and a lidar, to perform victim searches in post-disaster scenarios. Exploration is done not by following a pre-planned path (as common methods) but by prioritising the areas most likely to harbour victims. To accomplish that task, both indirect search and next best view techniques have been used. When ARTU-R gets inside an unstructured and unknown environment, it selects the next exploration point from a series of candidates. This operation is performed by comparing, for each candidate, the distance to reach it, the unexplored space around it and the probability of a victim being in its vicinity. This probability value is obtained using a Random Forest, which processes the information provided by a convolutional neural network. Unlike other AI techniques, random forests are not black box models; humans can understand their decision-making processes. The system, once integrated, achieves speeds comparable to other state-of-the-art algorithms in terms of exploration, but concerning victim detection, the tests show that the resulting smart exploration generates logical paths—from a human point of view—and that ARTU-R tends to move first to the regions where victims are present.

Cascade & allocate: A cross-structure adversarial attack against models fusing vision and language

The data fusion systems between multiple modals have attracted a wide range of concerns about their adversarial robustness. Recent image captioning attacks lack cross-structure transferability against models with diverse structures. Therefore, the attackers demonstrate satisfactory performance only when they gain full or partial knowledge of target image captioning models. So far, the cross-structure transferability of multi-modal adversarial examples has not been thoroughly investigated. In this paper, a theorem is proposed to analyze the upper bound captioning adversarial transferability. We design a new transfer-based adversarial attack method (Cascade & Allocate) against image captioning models. Our method can be divided into two steps. The first step randomly selects a set of candidate models with diverse structures, and we use a momentum-like strategy to generate perturbations. This ‘model cascade’ step narrows the gap of the gradient directions between different image captioning models, therefore enhancing the cross-model transferability and generating model-agnostic adversarial perturbations. The second step utilizes the upper bound transferability theorem to allocate the weight of perturbations per model. This ‘weight allocate’ step enhances the weight of model with the most consistent gradient in all candidate models, which generates more transferable adversarial examples. In the experiments, we compare our approach with other captioning and ensemble-based attacks against five black-box models with different structures on MS COCO and Flickr-30k datasets. The adversarial examples exhibit state-of-the-art adversarial transferability against five black-box models with different structures. The results demonstrate that our approach is practical and generalizes well to a wide range of captioning models.

Predicting lung cancer risk using explainable artificial intelligence

Lung cancer is a lethal disease that claims numerous lives annually, and early detection is essential for improving survival rates. Machine learning has shown promise in predicting lung cancer risk, but the lack of transparency and interpretability in black-box models impedes the understanding of factors that contribute to risk. Explainable artificial intelligence (XAI) can overcome this limitation by providing a clear and understandable approach to machine learning. In this study, we will use a large patient record dataset to train an XAI-based model that considers various patient information, including lifestyle factors, clinical data, and medical history, for predicting lung cancer risk. We will use different XAI techniques, including decision trees, partial dependence plots, and feature importance, to interpret the model’s predictions. These methods will provide healthcare professionals with a transparent and interpretable framework for screening and treatment decisions concerning lung cancer risk.

Prediction-Based Power Consumption Monitoring of Industrial Equipment Using Interpretable Data-Driven Models

Efficient energy optimization and scheduling in industrial factories depend on accurate, reasonable, and real-time monitoring of equipment power consumption. However, the power prediction of industrial equipment requires a large number of process data, which could be inevitably contaminated by some imperfect data, due to the harsh environments. Monitoring or predicting equipment power consumption is usually not feasible using data-driven black-box models with imperfect data. This paper proposes a prediction-based approach for power consumption monitoring using an interpretable data-driven model. First, a data preprocessing method is used to remove outliers and fill in missing values. Then, a Volterra polynomial basis function (VPBF) model is built to predict equipment power consumption. This model decomposes power values into a series of basis functions consisting of input parameters. Moreover, to compensate for data dropouts during the power consumption monitoring process, a networked predictive monitoring system is also proposed. Finally, this paper presents two case studies based on actual production equipment in an industrial manufacturing factory. The results demonstrate that the proposed approach can achieve satisfactory monitoring accuracy and adaptation ability. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the problem of power consumption monitoring of industrial equipment in harsh production environments. A conventional solution is to predict the power consumption using data-driven black-box models, with limited feasibility and interpretability. This paper proposes a new power consumption monitoring approach, utilizing an interpretable data-driven model and a networked predictive method. This approach accurately reveals the transparent relationships between the output and input parameters. The power prediction result is consequently interpretable and more reasonable. Meanwhile, this approach actively compensates for data dropouts in the network, which can help operators real-time grasp the power consumption of equipment. Furthermore, this approach can be integrated into energy optimization systems as a basis of optimization and decision-making. Practical applications in an aluminum manufacturing company located in Guangdong Province demonstrate that this approach is feasible and applicable. Future research is to expand this approach further for energy optimization and scheduling.

An Improved Black-Box Model for Low-Current AC Arcs in Vacuum Interrupters

An Improved Black-Box Model for Low-Current AC Arcs in Vacuum Interrupters

Prototype-Based Interpretable Graph Neural Networks

Graph neural networks have proved to be a key tool for dealing with many problems and domains such as chemistry, natural language processing and social networks. While the structure of the layers is simple, it is difficult to identify the patterns learned by the graph neural network. Several works propose post-hoc methods to explain graph predictions, but few of them try to generate interpretable models. Conversely, the topic of the interpretable models is highly investigated in image recognition. Given the similarity between image and graph domains, we analyze the adaptability of prototype-based neural networks for graph and node classification. In particular, we investigate the use of two interpretable networks, ProtoPNet and TesNet, in the graph domain. We show that the adapted networks manage to reach better or higher accuracy scores than their respective black-box models and comparable performances with state-of-the-art self-explainable models. Showing how to extract ProtoPNet and TesNet explanations on graph neural networks, we further study how to obtain global and local explanations for the trained models. We then evaluate the explanations of the interpretable models by comparing them with post-hoc approaches and self-explainable models. Our findings show that the application of TesNet and ProtoPNet to the graph domain produces qualitative predictions while improving their reliability and transparency.

Advancing Educational Insights: Explainable AI Models for Informed Decision-Making

Abstract: Over the past two decades, the integration of machine learning (ML) techniques within educational frameworks has garnered significant attention. However, despite its widespread adoption, there remains a dearth of research focusing on developing AI systems with a core emphasis on interpretability and explainability. This paper seeks to bridge this gap by proposing an advanced framework that not only predicts students' performance accurately but also offers reliable and interpretable results tailored for career counseling. The framework merges the concepts of ML and Explainable AI (XAI) to address the complexities of career counseling in educational settings. Drawing inspiration from educational data mining, the framework aims to provide insights conducive to students' career growth and decision-making processes. By incorporating MLbased White and Black Box models, the approach analyzes a comprehensive educational dataset comprising academic and employability attributes crucial for job placements and skill development. To enhance interpretability, the framework leverages the NGBoost algorithm, known for its efficiency in prediction modeling. Additionally, it integrates Local Interpretable Modelagnostic Explanations (LIME) and Shapley Additive Explanations (SHAP) methods for providing both local and global explanations of model predictions, ensuring transparency and comprehensibility. Through a series of use cases, the paper showcases the applicability and effectiveness of the framework in providing actionable insights to educators and students alike. In conclusion, this research contributes to the advancement of career counseling in educational contexts by offering a robust and interpretable ML-based framework. By providing transparent insights into students' academic performance and career prospects, the approach facilitates informed decision-making and supports personalized guidance for optimal career trajectories.

Heuristic model predictive control implementation to activate energy flexibility in a fully electric school building

This paper presents a heuristic model predictive control (MPC) methodology to activate energy flexibility in fully electric school buildings in cold climates to reduce electricity demand during peak demand periods of the electric grid. To streamline the implementation of MPC, the proposed approach employs grey-box archetypes, a clustering of weather conditions to identify typical scenarios and a limited number of possible setpoint profiles. A data-driven grey-box approach is used to create archetype models for different thermal zones in a typical school building; this approach enables rapid development and requires much less calibration data than black-box models. A third-order resistance-capacitance thermal network for zones with convective heating and a fourth-order model for zones with radiant floor heating are developed and calibrated using measured data from an all-electric school building in Québec, Canada. The weather data are clustered into several categories, representing different weather conditions (6 clusters representing two ambient temperature ranges and three solar radiation ranges). The heuristic MPC strategy uses predefined optimal setpoint profiles for each cluster and weather prediction one day ahead to shift the building load from on-peak to off-peak hours. For each heuristic MPC scenario, the model runs a simulation using forecast weather data to quantify and activate energy flexibility in response to grid requirements. The developed MPC framework was implemented in the school used as the case study. Ten classrooms are investigated, with six using the MPC and four as a reference case with the reactive control system and default zone setpoint profiles. Results indicate that the school building can provide between 47% to 95% energy flexibility (load shifting relative to reference) during on-peak hours and up to 44% electricity cost reduction while satisfying acceptable temperature constraints. By implementing the proposed MPC, energy flexibility of 32 W/m2 of floor area for the zones with a convective heating system and 65 W/m2 of floor area for the zones with radiant heating can be achieved during a demand response event. The proposed strategy can be generalized and replicated in other school buildings.

Identification method of all-operating-point admittance model for wind farms considering frequency-coupling characteristics

Admittance identification is an effective tool to solve the issue of black-box admittance modeling in wind farms. However, due to the large operating range of wind farms, it is difficult to accurately identify the admittance model of wind farms under all operating conditions, which limits the research on wideband oscillation caused by the interaction between wind farms and the power grid. This paper analyses the characteristics of the frequency-coupling admittance for wind farms. It is found that the power-sharing ratio of wind turbine generators (WTGs) affects the admittance characteristic of the wind farm, which leads to a huge number of operating points for the wind farm. The all-operating-point (AOP) admittance model of wind farms considering frequency-coupling characteristics is hard to directly identify. Aiming for this problem, this paper proposes to first identify the AOP-admittance model for WTGs. Then, based on the operating points of each WTG and the network structure, the AOP-admittance model of the wind farm is obtained via the admittance aggregation and model reduction method. This proposed method solves the problem of black-box admittance modeling and addresses the problem of dimension disaster in admittance modeling for wind farms. The identified model of the wind farm can be used to accurately assess the stability of the wind power integration system under various operating conditions. Finally, the effectiveness of the proposed method is verified through the admittance measurement and stability analysis of the wind farm.

Optimising machine learning prediction of minimum inhibitory concentrations in Klebsiella pneumoniae.

Minimum Inhibitory Concentrations (MICs) are the gold standard for quantitatively measuring antibiotic resistance. However, lab-based MIC determination can be time-consuming and suffers from low reproducibility, and interpretation as sensitive or resistant relies on guidelines which change over time. Genome sequencing and machine learning promise to allow in silico MIC prediction as an alternative approach which overcomes some of these difficulties, albeit the interpretation of MIC is still needed. Nevertheless, precisely how we should handle MIC data when dealing with predictive models remains unclear, since they are measured semi-quantitatively, with varying resolution, and are typically also left- and right-censored within varying ranges. We therefore investigated genome-based prediction of MICs in the pathogen Klebsiella pneumoniae using 4367 genomes with both simulated semi-quantitative traits and real MICs. As we were focused on clinical interpretation, we used interpretable rather than black-box machine learning models, namely, Elastic Net, Random Forests, and linear mixed models. Simulated traits were generated accounting for oligogenic, polygenic, and homoplastic genetic effects with different levels of heritability. Then we assessed how model prediction accuracy was affected when MICs were framed as regression and classification. Our results showed that treating the MICs differently depending on the number of concentration levels of antibiotic available was the most promising learning strategy. Specifically, to optimise both prediction accuracy and inference of the correct causal variants, we recommend considering the MICs as continuous and framing the learning problem as a regression when the number of observed antibiotic concentration levels is large, whereas with a smaller number of concentration levels they should be treated as a categorical variable and the learning problem should be framed as a classification. Our findings also underline how predictive models can be improved when prior biological knowledge is taken into account, due to the varying genetic architecture of each antibiotic resistance trait. Finally, we emphasise that incrementing the population database is pivotal for the future clinical implementation of these models to support routine machine-learning based diagnostics.

Research on the Transformation Efficiency of Scientific and Technological Achievements of Minis-try-affiliated Universities Based on Network DEA

Previous studies on the transformation of scientific and technological achievements in universities were based on the black box model, and the results obtained were not comprehensive enough. This paper divides the transformation of scientific and technological achievements of universities into the knowledge research and development stage and the value transformation stage. The two-stage network DEA model is used to study the transformation efficiency of scientific and technological achievements of 55 subordinate universities based on the data from 2009 to 2017.The research in this paper finds that the overall efficiency of the transformation of scientific and technological achievements in Chinese universities is relatively stable, and the efficiency of the knowledge research and development stage is relatively high, but the efficiency of the value realization stage is seriously low, resulting in a low level of overall efficiency."985" engineering colleges and "double first-class" colleges have higher input and output efficiency, and cluster analysis is used to classify and summarize the technological transformation of each college. Finally, corresponding development suggestions are put forward.

Learning Performance Maximizing Ensembles with Explainability Guarantees

In this paper we propose a method for the optimal allocation of observations between an intrinsically explainable glass box model and a black box model. An optimal allocation being defined as one which, for any given explainability level (i.e. the proportion of observations for which the explainable model is the prediction function), maximizes the performance of the ensemble on the underlying task, and maximizes performance of the explainable model on the observations allocated to it, subject to the maximal ensemble performance condition. The proposed method is shown to produce such explainability optimal allocations on a benchmark suite of tabular datasets across a variety of explainable and black box model types. These learned allocations are found to consistently maintain ensemble performance at very high explainability levels (explaining 74% of observations on average), and in some cases even outperform both the component explainable and black box models while improving explainability.

Faithful Model Explanations through Energy-Constrained Conformal Counterfactuals

Counterfactual explanations offer an intuitive and straightforward way to explain black-box models and offer algorithmic recourse to individuals. To address the need for plausible explanations, existing work has primarily relied on surrogate models to learn how the input data is distributed. This effectively reallocates the task of learning realistic explanations for the data from the model itself to the surrogate. Consequently, the generated explanations may seem plausible to humans but need not necessarily describe the behaviour of the black-box model faithfully. We formalise this notion of faithfulness through the introduction of a tailored evaluation metric and propose a novel algorithmic framework for generating Energy-Constrained Conformal Counterfactuals that are only as plausible as the model permits. Through extensive empirical studies, we demonstrate that ECCCo reconciles the need for faithfulness and plausibility. In particular, we show that for models with gradient access, it is possible to achieve state-of-the-art performance without the need for surrogate models. To do so, our framework relies solely on properties defining the black-box model itself by leveraging recent advances in energy-based modelling and conformal prediction. To our knowledge, this is the first venture in this direction for generating faithful counterfactual explanations. Thus, we anticipate that ECCCo can serve as a baseline for future research. We believe that our work opens avenues for researchers and practitioners seeking tools to better distinguish trustworthy from unreliable models.

UFDA: Universal Federated Domain Adaptation with Practical Assumptions

Conventional Federated Domain Adaptation (FDA) approaches usually demand an abundance of assumptions, which makes them significantly less feasible for real-world situations and introduces security hazards. This paper relaxes the assumptions from previous FDAs and studies a more practical scenario named Universal Federated Domain Adaptation (UFDA). It only requires the black-box model and the label set information of each source domain, while the label sets of different source domains could be inconsistent, and the target-domain label set is totally blind. Towards a more effective solution for our newly proposed UFDA scenario, we propose a corresponding methodology called Hot-Learning with Contrastive Label Disambiguation (HCLD). It particularly tackles UFDA's domain shifts and category gaps problems by using one-hot outputs from the black-box models of various source domains. Moreover, to better distinguish the shared and unknown classes, we further present a cluster-level strategy named Mutual-Voting Decision (MVD) to extract robust consensus knowledge across peer classes from both source and target domains. Extensive experiments on three benchmark datasets demonstrate that our method achieves comparable performance for our UFDA scenario with much fewer assumptions, compared to previous methodologies with comprehensive additional assumptions.

Depression Detection via Capsule Networks with Contrastive Learning

Depression detection is a challenging and crucial task in psychological illness diagnosis. Utilizing online user posts to predict whether a user suffers from depression seems an effective and promising direction. However, existing methods suffer from either poor interpretability brought by the black-box models or underwhelming performance caused by the completely separate two-stage model structure. To alleviate these limitations, we propose a novel capsule network integrated with contrastive learning for depression detection (DeCapsNet). The highlights of DeCapsNet can be summarized as follows. First, it extracts symptom capsules from user posts by leveraging meticulously designed symptom descriptions, and then distills them into class-indicative depression capsules. The overall workflow is in an explicit hierarchical reasoning manner and can be well interpreted by the Patient Health Questionnaire-9 (PHQ9), which is one of the most widely adopted questionnaires for depression diagnosis. Second, it integrates with contrastive learning, which can facilitate the embeddings from the same class to be pulled closer, while simultaneously pushing the embeddings from different classes apart. In addition, by adopting the end-to-end training strategy, it does not necessitate additional data annotation, and mitigates the potential adverse effects from the upstream task to the downstream task. Extensive experiments on three widely-used datasets show that in both within-dataset and cross-dataset scenarios our proposed method outperforms other strong baselines significantly.

Attacking Transformers with Feature Diversity Adversarial Perturbation

Understanding the mechanisms behind Vision Transformer (ViT), particularly its vulnerability to adversarial perturbations, is crucial for addressing challenges in its real-world applications. Existing ViT adversarial attackers rely on labels to calculate the gradient for perturbation, and exhibit low transferability to other structures and tasks. In this paper, we present a label-free white-box attack approach for ViT-based models that exhibits strong transferability to various black-box models, including most ViT variants, CNNs, and MLPs, even for models developed for other modalities. Our inspiration comes from the feature collapse phenomenon in ViTs, where the critical attention mechanism overly depends on the low-frequency component of features, causing the features in middle-to-end layers to become increasingly similar and eventually collapse. We propose the feature diversity attacker to naturally accelerate this process and achieve remarkable performance and transferability.